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Quantitative  modeling  is  a  powerful  tool  with  which  to  study  human  performance  and  behavior.  Despite  the 
proven  usefulness  of  modeling,  it  remains  under-utilized  by  the  research  community,  in  part  because 
models  can  be  difficult  to  implement  and  data  sets  difficult  to  obtain  or  generate.  The  purpose  of  this 
project  was  to  promote  cognitive  modeling  by  creating  an  online  repository  for  the  submission  and 
acquisition  (i.e.,  downloading)  of  computational  models  of  cognition  and  corresponding  data  sets  to  model. 
The  repository  was  created  (cmr.osu.edu).  All  site  visitors  can  peruse  and  download  content.  Those  who 
register  can  upload  content. 
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Final  Report 


The  purpose  of  this  project  was  to  create  the  Cognitive  Modeling  Repository  (CMR),  an  online 
repository  for  the  submission  and  dissemination  of  computational  models  of  cognition  and  data 
sets  with  which  to  model.  This  objective  has  been  met.  The  repository  can  be  found  at 
cmr.osu.edu. 

Summary  of  Repository  Creation 

Design  of  the  repository  went  through  multiple  revisions  to  maximize  ease  of  navigation  and  use, 
ensure  browser  compatibility,  and  ensure  stability.  The  process  for  submitting  a  model  or  a  data 
set  has  been  designed  to  be  simple.  Currently,  only  models  and  data  sets  that  have  appeared  in 
peer-reviewed  publications  can  be  submitted.  Once  the  user  has  compiled  the  necessary  fdes, 
submission  takes  no  more  than  ten  minutes.  Submitters  must  agree  to  the  terms  of  the  Creative 
Commons  Zero  (CCO)  license,  which  grants  others  use  of  the  model  or  data  set  free  of  copyright 
restrictions  to  the  extent  allowed  by  law.  It  is  the  formalized  equivalent  of  the  requirement  by 
some  publishers  that  authors  make  their  model  and  data  available  to  other  researchers  upon 
request,  once  a  manuscript  is  published. 

The  browsing  of  models  and  data  sets  stored  in  the  repository  is  similarly  simple  in  functionality. 
The  user  is  provided  with  a  subset  of  the  most  potentially  useful  information  in  a  visual  layout 
that  minimizes  the  amount  of  searching  and  reduces  the  number  of  mouse  clicks  necessary  to 
find  information. 

Stress  tests  (usability  and  bug-finding)  were  performed  with  groups  of  individuals  to  evaluate 
and  improve  the  robustness  of  the  web  site.  There  are  4.5  terabytes  of  disk  space  available  for 
model  and  data  storage.  The  server  containing  the  repository  is  located  in  the  Department  of 
Psychology  at  Ohio  State  University. 

The  Challenge  of  Populating  the  Repository 

One  goal  of  the  project  was  to  seed  the  repository  with  30  models  and  data  sets  prior  to  opening 
it  the  public.  These  submissions  were  to  be  well-known  models  and  data  sets  in  the  field,  with 
the  idea  that  they  would  act  as  a  catalyst  in  encouraging  researchers  in  the  cognitive  science 
community  to  not  only  download  models  and  data  sets,  but  also  make  contributions  as  well.  The 
long-term  goal  was  that  the  repository  would  evolve  into  a  regularly  used  community  resource, 
much  like  a  journal. 

Although  there  was  widespread  and  virtually  unanimous  enthusiasm  for  the  repository,  including 
praises  from  many  individuals  in  AFRL  and  AFOSR  (this  feedback  was  obtained  at  conference 
presentations  promoting  the  web  site),  the  Pis  were  able  to  obtain  contributions  of  only  eight 
models  and  a  half-dozen  data  sets.  Many  attempts  were  made  to  increase  contributions,  including 
repeated  requests  to  do  so,  suggesting  other  lab  personnel  (e.g.,  graduate  students)  make  the 
contribution,  and  offering  to  make  the  submission  on  behalf  of  the  author.  These  strategies  have 
been  unsuccessful  thus  far,  and  highlight  the  difficulty  of  making  this  task  a  priority  for  busy 
scholars. 


Suggested  Next  Steps  in  Repository  Development 


Data  repositories  are  the  norm  in  many  fields  (e.g.,  astronomy,  computer  science,  biology, 
chemistry).  They  are  an  easy  method  by  which  to  promote  data  sharing  and  benefit  the  science 
by  reducing  duplication  and  facilitating  knowledge  accumulation.  Establishment  of  CMR  is  a 
necessary  first  step  to  ensure  cognitive  modeling,  and  cognitive  science  more  generally,  follows 
in  this  same  path.  Critical  to  CMR’s  success  is  requiring  contributions  from  modelers.  This  has 
been  achieved  in  other  disciplines  by  journal  publishers  making  submission  (e.g.,  of  a  data  set)  a 
requirement  prior  to  manuscript  publication.  To  convince  publishers  to  make  such  a  change  in 
publication  policy  would  require  legwork.  One  strategy  would  be  to  fonn  an  executive 
committee  of  respected  scholars  in  modeling  who  are  sympathetic  to  the  repository  and  have 
committee  representatives  make  the  case  to  publishers.  This  has  worked  in  some  fields.  Support 
from  funding  agencies,  such  as  AFOSR  (NIH  and  NSF  have  funded  repositories)  would  help. 
AFOSR  could  also  implement  a  similar  policy  by  making  model  and  data  set  submission  a 
requirement  for  research  funding. 

On  a  more  pragmatic  level,  future  enhancements  of  the  site  should  address  two  issues:  expanding 
content  and  platform  longevity.  CMR  should  contain  educational  materials  as  another  means  of 
promoting  and  encouraging  modeling.  This  was  a  goal  of  the  original  proposal  and  the  Pis  began 
to  assemble  some  materials,  but  they  are  insufficient  and  incomplete  to  add  to  the  current  site. 
What  is  needed  are  written  tutorials  and  video  lectures  on  a  range  of  modeling  styles.  The 
availability  of  such  content  would  attract  a  wider  range  of  users  and  has  the  potential  to  increase 
interest  in  modeling  by  students  and  researchers  alike. 

The  second  issue  pertains  only  to  the  software  used  for  model  and  data  set  submission.  It  is 
written  in  Adobe  Flash,  which  was  chosen  because  it  is  in  wide  use  and  applications  written  in 
Flash  are  browser  and  platform  independent.  In  December  of  201 1,  Adobe  announced  it  was 
abandoning  Flash  in  favor  of  HTMF5.  The  implications  of  this  for  CMR  are  that,  although  the 
submission  pages  will  work  fine  for  a  couple  of  years  (all  browsers  will  continue  to  support  flash 
for  the  time  being),  the  software  will  eventually  become  obsolete.  The  sooner  this  part  of  CMR 
is  rewritten  in  HTMF5,  the  better. 


